Hierarchical anomaly localization and prioritization

ABSTRACT

Methods, apparatus and articles of manufacture for hierarchical anomaly localization and prioritization are disclosed. An example method disclosed herein comprises obtaining reported status for a plurality of nodes of a hierarchical topology, the reported status for a particular node being at least one of normal, abnormal or indeterminate, and determining a subset of root cause abnormal nodes, a root cause abnormal node included in the subset of root cause abnormal nodes having a total number of abnormal direct descendent nodes and indeterminate direct descendent nodes that is greater than a number of normal direct descendent nodes of the root cause abnormal node.

FIELD OF THE DISCLOSURE

This disclosure relates generally to anomaly processing and, moreparticularly, to hierarchical anomaly localization and prioritization.

BACKGROUND

In recent years, Internet Service Providers (ISPs) have been rolling outa wide range of value added services beyond basic connectivity, such asweb hosting, content distribution network (CDN) services, databaseservices, gaming services, cloud computing, e-commerce server hosting,etc. In many cases, customers access these value added services via anInternet connection and, as such, customers can be dispersed over a widegeographic area. Additionally, the value added services provided by anISP are often hosted in geographically distributed data centers, whichmay be co-located with an ISP's different Points of Presence (PoPs).Detecting and localizing end-to-end performance issues for suchwide-area services can be useful to an ISP operator for achievingdesired end user service quality, such as by enabling fast serviceimpairment detection and flexible mitigation control. Existingtechniques for detecting and localizing performance issues and othernetwork anomalies can involve using active probes placed strategicallyin an ISP's wide-area network that inject probe packets into the networkto enable detection of anomalies associated with the locations of theactive probes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is block diagram of an example system to perform hierarchicalanomaly localization and prioritization.

FIG. 2 is a block diagram of a first example hierarchical topology forwhich the example system of FIG. 1 can perform hierarchical anomalylocalization and prioritization.

FIG. 3 is a block diagram of a second example hierarchical topology forwhich the example system of FIG. 1 can perform hierarchical anomalylocalization and prioritization.

FIG. 4 is a block diagram of an example anomaly localizer that can beused to implement the example system of FIG. 1.

FIG. 5 illustrates an example operation of the example anomaly localizerof FIG. 1.

FIG. 6 is a block diagram of an example anomaly prioritizer that can beused to implement the example system of FIG. 1.

FIG. 7 is a flowchart representative of example machine readableinstructions that may be executed to implement the example system ofFIG. 1.

FIG. 8 is a flowchart representative of example machine readableinstructions that may be executed to implement the example anomalylocalizer of FIG. 4.

FIG. 9 is a flowchart representative of example machine readableinstructions that may be executed to implement a greedy algorithm foruse in implementing the example anomaly localizer of FIG. 4 and/or theexample machine readable instructions of FIG. 8.

FIG. 10 depicts example pseudocode representative of example machinereadable instructions that may be used to implement the flowchart ofFIG. 9.

FIG. 11 is a flowchart representative of example machine readableinstructions that may be executed to implement the example anomalyprioritizer of FIG. 6.

FIG. 12 is a block diagram of an example processing system that mayexecute the example machine readable instructions represented by FIGS.7-10 and/or 11 to implement the example system of FIG. 1, the exampleanomaly localizer of FIG. 4 and/or the example anomaly prioritizer ofFIG. 6.

DETAILED DESCRIPTION

Methods, apparatus and articles of manufacture for hierarchical anomalylocalization and prioritization are disclosed herein. An example methoddisclosed herein includes obtaining reported status for a plurality ofnodes of a hierarchical topology. The reported status for a particularnode can be at least one of normal, abnormal or indeterminate. Theexample method also includes determining a subset of root cause abnormalnodes that covers a set of abnormal nodes in the hierarchical topologyindicated by the reported status. In some examples, a root causeabnormal node that is included in the subset of root cause abnormalnodes has a total number of abnormal direct descendent nodes andindeterminate direct descendent nodes that is greater than a number ofnormal direct descendent nodes of the root cause abnormal node.

In some examples, to determine the subset of root cause abnormal nodes,the method further includes selecting a set of candidate root causeabnormal nodes from the set of abnormal nodes in the hierarchicaltopology. For example, a candidate root cause abnormal node can be anyabnormal node that has a respective number of abnormal direct descendentnodes and indeterminate direct descendent nodes that is greater than arespective number of normal direct descendent nodes of the candidateroot cause abnormal node. After selecting the set of candidate rootcause abnormal nodes from the set of abnormal nodes, the example methodthen determines the subset of root cause abnormal nodes from the set ofcandidate root cause abnormal nodes. For example, for each respectivecandidate root cause abnormal node, the method can determine arespective subset of the set of abnormal nodes in the hierarchicaltopology covered by the respective candidate root cause abnormal node. Aparticular candidate root cause abnormal node covers an abnormal nodeif, for example, the abnormal node is the particular candidate rootcause abnormal node itself, or the abnormal node is a descendent of theparticular candidate root cause abnormal node, or the abnormal node isan ancestor of the particular candidate root cause abnormal node. Insome examples, the method employs a greedy technique involving selectinga first candidate root cause abnormal node for inclusion in the subsetof root cause abnormal nodes if the first candidate root cause abnormalnode covers a largest subset of the set of abnormal nodes in thehierarchical topology. The method then continues to iteratively select anext candidate root cause abnormal node covering the next largest subsetof the set of abnormal nodes for inclusion in the subset of root causeabnormal nodes.

An example apparatus disclosed herein includes an example candidateselector (e.g., implemented by a first processor) to select a set ofcandidate root cause abnormal nodes from a set of abnormal nodesidentified in a hierarchical topology. As noted above, a particularcandidate root cause abnormal node may be any abnormal node that has atotal number of abnormal direct descendent nodes and indeterminatedirect descendent nodes that is greater than a number of normal directdescendent nodes of the particular candidate root cause abnormal node.The example apparatus also includes an example root cause determiner(e.g., implemented by at least one of the first processor or a secondprocessor) to determine a set of root cause abnormal nodes from the setof candidate root cause abnormal nodes selected by the candidateselector. The set of root cause abnormal nodes is to cover the set ofabnormal nodes identified in the hierarchical topology. For example, theroot cause determiner can iteratively select candidate root causeabnormal nodes for inclusion in the subset of root cause abnormal nodesbased on sizes of the subsets of abnormal nodes in the hierarchicaltopology covered by the respective candidate root cause abnormal nodes.

In some examples, the apparatus further includes a prioritizer todetermine a size (e.g., degree, severity, etc.) of a respectiveabnormality associated with each root cause abnormal node included inthe subset of root cause abnormal nodes. The prioritizer can alsodetermine a scope (e.g., number of end users affected, geographic regionaffected, etc.) for the respective abnormality associated with each rootcause abnormal node included in the subset of root cause abnormal nodes.The prioritizer can further rank the root cause abnormal nodes includedin the subset of root cause abnormal nodes based on size and/or scope.

In the preceding examples, the nodes of the hierarchical topology caninclude, for example, physical, logical and/or geographical networkelements at different hierarchical levels in a network. For example, ina network employing the border gateway protocol (BGP), the nodes caninclude routers, autonomous systems (ASes), AS paths, BGP prefixes,etc., and/or the particular geographic location served by these physicaland logical network elements. As another example, in a 3rd generationmobile network, the nodes can include serving GPRS supports nodes(SGSNs, where GPRS refers to the general packet radio service), radionetwork controllers (RNCs), etc., and/or the particular markets andsub-markets served by these physical and logical network elements.

As noted above, existing techniques for detecting and localizingperformance issues and other network anomalies can involve using activeprobes placed at particular locations in an ISP's wide-area network. Theactive probes inject probe packets into the network to enable detectionof anomalies associated with the locations of the active probes. Suchactive probe techniques are, therefore, limited in that networkanomalies can be localized only to the particular locations at which theactive probes are placed. In contrast, the example methods, apparatusand articles of manufacture disclosed herein can be used to localizenetwork anomalies at any specified or identified physical, logical,geographic, etc., node in a network hierarchical topology. The examplemethods, apparatus and articles of manufacture disclosed herein can alsolocalize anomalies to determine root cause abnormal node(s) usingpassive, as well as active, network monitoring. Furthermore, in someexamples, the methods, apparatus and articles of manufacture disclosedherein can prioritize the root cause abnormal node based on one or morecriteria, such as anomaly size, scope, etc.

Turning to the figures, a block diagram of an example system 100 toperform hierarchical anomaly localization and prioritization asdisclosed herein is illustrated in FIG. 1. The example system 100includes an example anomaly detector 105 to detect anomalies associatedwith one or more nodes of a hierarchical topology. In the illustratedexample, the anomaly detector 105 is configured by an example userworkstation 110. The user workstation 110 may be implemented by any typeof workstation for use by any type of user, operator, administrator,etc. For example, the user workstation 110 may be implemented by acomputer workstation, a server, a desktop computer, a notebook computer,a tablet computer, a personal digital assistant (PDA), a smartphone, theexample processing system 1200 illustrated in FIG. 12, etc., or anycombination thereof.

In the illustrated example, the user workstation 110 can be used toconfigure a hierarchical topology 115 for which anomalies are to bedetected by the anomaly detector 105. Example hierarchical topologies115 for which the anomaly detector 105 may be configured to performanomaly detection are illustrated in FIGS. 2-3, which are described ingreater detail below. Other configuration information 120, such asanomaly criteria, measurement parameters, etc., for the anomaly detector105 may additionally or alternatively be configured by the userworkstation 110. In some examples, one or more of the hierarchicaltopology 115 and/or other configuration information 120 may bepredetermined (e.g., initialized, hard-coded, etc.) in the anomalydetector 105.

Example hierarchical topologies 200 and 300 that may correspond to thehierarchical topology 115 configured by the user workstation 110 areillustrated in FIGS. 2 and 3, respectively. The example hierarchicaltopology 200 of FIG. 2 corresponds to a BGP-based network implementingan intelligent content distribution service (ICDS). The examplehierarchical topology 300 of FIG. 3 corresponds to a 3rd generationmobile communications network. The hierarchical topologies 200 and 300include nodes representing physical, logical, geographical, etc.,elements of the network aggregated at different levels in the networkhierarchy. For a particular node in the hierarchical topology 200 or300, its descendent node(s) are the node(s) connected to the particularnode by one or more edges and located at one or more lower hierarchicallevels of the hierarchical topology, whereas its ancestor node(s) arethe node(s) connected to the particular node by one or more edges andlocated at one or more higher hierarchical levels of the hierarchicaltopology 200/300.

For example, in the hierarchical topology 200 of FIG. 2, the lowestlevels of the hierarchy include nodes 205 and 210 representing differentBGP prefixes of the BGP-based network, and nodes 215 and 220representing different geographic regions (e.g., cities) served by theBGP-based network. In the illustrated example, the BGP prefix nodes 205and 210 are aggregated into one or more ancestor nodes 225 and 230representing the AS paths associated with these BGP prefixes. Forexample, the node 225 represents the AS path associated with the BGPprefixes represented by the descendent nodes 205 and 210.

In the example hierarchical topology 200, the BGP prefix nodes 205 and210, the city nodes 215 and 220, and the AS path nodes 225 and 230 areaggregated into one or more ancestor nodes 235, 240, 245 and 250representing the autonomous systems associated with each of thesedescendent nodes. For example, the node 240 represents an originalautonomous system associated with the BGP prefix nodes 205/210 and thecity nodes 215/220, whereas the node 250 represents a next-hopautonomous system associated with the AS path nodes 225 and 230. Thehierarchical topology 200 of the illustrated example further aggregatesthe autonomous system nodes 235-250 into one or more ancestor nodes 255and 260 representing the egress routers (e.g., provider edge routers)associated with these autonomous systems. The egress router nodes 255and 260 are aggregated into a top-level node 265 of the hierarchicaltopology 200 representing the intelligent content distribution serviceprovided by the BGP-based network.

Turning to the example hierarchical topology 300 of FIG. 3, the lowestlevel of the hierarchy include nodes 305 and 310 representing differentcell sites of the mobile network. In the illustrated example, the sitenodes 305 and 310 are aggregated into one or more ancestor nodes 315 and320 representing the RNCs associated with these cell sites. For example,the node 320 represents the RNC associated with the cell sitesrepresented by the descendent nodes 305 and 310. In the hierarchicaltopology 300, the RNC nodes 315 and 320 are aggregated into one or moreancestor nodes representing the SGSNs associated with these RNCs. Forexample, the node 330 represents the SGSN associated with the RNCsrepresented by the descendent nodes 315 and 320. The SGSN nodes 325 and330 are further aggregated into one or more ancestor nodes 335 and 340representing the sub-markets associated with these SGSNs. For example,the node 335 represents the sub-market associated with the SGSNsrepresented by the descendent nodes 325 and 330. The sub-market nodes335 and 340 are aggregated into a top-level node 345 of the hierarchicaltopology 300 representing the market served by the mobile network.

Returning to the illustrated example of FIG. 1, the anomaly detector 105included in the system 100 can perform any type of anomaly detectionprocessing on measurements 125 obtained for the hierarchical topology115 configured by the user workstation 110 to detect anomaliesassociated with one or more nodes of the hierarchical topology 115. Forexample, the anomaly detector 105 can be configured to processend-to-end round-trip-time (RTT) measurements for communications betweenan end-user and a server providing a service, such as one or more of thevalue-added services mentioned above. In such an example, thehierarchical topology 115 can correspond to the network providing theservice(s), and the other configuration information 120 can specifyparameters for (1) processing the RTT measurements (e.g., averaging overtime, segmenting/binning over time, etc.), (2) aggregating the RTTmeasurements at different nodes in the hierarchical topology 115, (3)determining whether the RTT measurements are indicative of an anomaly(e.g., based on exceeding a specified threshold, deviating from aspecified RTT range, etc.), etc.

The anomaly detector 105 of the illustrated example processes themeasurements 125 to provide a reported status 130 for the nodes in thehierarchical topology 115. For example, a node associated with adetected anomaly may have a reported status of “abnormal” to indicatethat the node is an abnormal node, whereas a node that is not associatedwith a detected anomaly may have a reported status of “normal” toindicate that the node is a normal node. In some examples, the anomalydetector 105 can also support reporting a status of “indeterminate” fora particular node to indicate that the results of anomaly detection forthe particular node were inconclusive (e.g., due to insufficientmeasurements for that particular node and/or for the associated level ofthe hierarchical topology 115).

The example system 100 of FIG. 1 includes an example anomaly localizer135 to receive or otherwise obtain the reported status from the anomalydetector 105 and to locate, or localize, the anomalies in thehierarchical topology 115. As such, in the illustrated example, theanomaly localizer 135 is also configured by the user workstation 110with the hierarchical topology 115. The example anomaly localizer 135obtains the reported status 130 from the anomaly detector 105 thatindicates whether each node in the hierarchical topology 115 is anabnormal node, a normal node, or an indeterminate node. For example, thereported status 130 obtained from the anomaly detector 105 may include avalue of “1” for an abnormal node, a value of “0” for a normal node, anda value of “−1” for an indeterminate node.

In the illustrated example, the anomaly localizer 135 uses the reportedstatus from the anomaly detector 105 to localize the anomalies detectedby the anomaly detector 105 in the hierarchical topology 115 into asubset of root cause abnormal nodes of the hierarchical topology 115.For example, in the hierarchical topologies 200 and/or 300 representingcommunication networks, a single underlying network event (e.g., such asa link failure) may manifest itself as anomalies at different hierarchylevels. As an illustrative example, in the hierarchical topology 200,assume that an underlying network event has caused an increase of RTTfor all user requests associated with a particular BGP prefix. In suchan example, the anomaly detector 105 can detect the RTT anomaly for thenode of the hierarchical topology 200 corresponding to this BGP prefix.Additionally, due to the nature of BGP routing, these user requests mayshare the same origin AS and AS path. If the user requests from theabnormal BGP prefix experiencing the anomalous RTT dominate other userrequests associated with the same origin AS or AS path, the anomalydetector 105 may also detect RTT anomalies for the nodes of thehierarchical topology 200 corresponding to this origin AS and the ASpath. In such an example, the anomaly localizer 135 can localize theanomaly to a root cause abnormal node corresponding to the noderepresenting the BGP prefix experiencing the RTT anomaly. As a converseexample, if a network event has impacted a particular AS path andcreated a service anomaly, its associated descendent(s) in thehierarchical topology 200, such as the particular BGP prefixesassociated with this AS path, would experience service anomalies aswell. In such an example, the anomaly localizer 135 localizes theanomaly to a root cause abnormal node corresponding to the noderepresenting the AS path experiencing the anomaly.

More generally, in the example system 100, the anomaly localizer 135obtains the reported status 130 from the anomaly detector 105 thatidentifies a set of abnormal nodes that are associated with detectedanomalies at one or more hierarchical levels of a specified hierarchicaltopology 115. The anomaly localizer 135 then processes this set ofreported abnormal nodes to determine and report a subset of root causeabnormal nodes 140, such as a smallest subset of the reported abnormalnodes, that can account for (or cover) all, or a particular portion of,the set of reported abnormal nodes associated with anomalies detected bythe anomaly detector 105. In the illustrated example, the anomalylocalizer 135 reports the subset of root cause abnormal nodes 140 to theuser workstation 110 for display and/or other post-processing. Anexample implementation of the anomaly localizer 135 is illustrated inFIG. 4, which is described in greater detail below.

The system 100 of FIG. 1 additionally includes an example anomalyprioritizer 145 to also receive or otherwise obtain the subset of rootcause abnormal nodes 140 reported by the anomaly localizer 135. In theillustrated example, the anomaly prioritizer 145 prioritizes the rootcause abnormal nodes included in the subset of root cause abnormal nodes140. For example, the anomaly prioritizer 145 can employ a rankingfunction to provide a ranking 150 of the root cause abnormal nodesincluded in the subset of root cause abnormal nodes 140. In someexamples, the ranking function is based on one or more ranking factors,such as the relative sizes (e.g., magnitude, degree, etc.) of theanomalies associated with the different root cause abnormal nodes, therelative scopes (e.g., number of users affected, number of nodesaffected, etc.) of the anomalies associated with the different rootcause abnormal nodes, etc. In the illustrated example, the anomalyprioritizer 145 reports the ranking 150 of the root cause abnormal nodesto the user workstation 110 for display and/or other post-processing. Anexample implementation of the anomaly prioritizer 145 is illustrated inFIG. 6, which is described in greater detail below.

A block diagram of an example implementation of the anomaly localizer135 of FIG. 1 is illustrated in FIG. 4. The example anomaly localizer135 of FIG. 4 localizes anomaly events in a hierarchical topology 115.The anomaly events are identified in the reported status 130 (e.g.,obtained from the anomaly detector 105) as a set of abnormal nodesincluded in the hierarchical topology 115. In the illustrated, theanomaly localizer 135 localizes the anomaly events by determining asubset of root cause abnormal nodes 140, such as a smallest number ofthe reported abnormal nodes, that cover (or account for) all, or aparticular portion of, the set of reported abnormal nodes included inthe reported status 115. In some examples, the anomaly localizer 135performs anomaly localization based on the following mathematicalproblem formulation.

For example, assume that the hierarchical topology 115 for which anomalydetection and localization is to be performed can be represented by adirected acyclic graph (DAG). Examples of such DAGs include, but are notlimited to, the example hierarchical topologies 200 and 300 illustratedin FIGS. 2 and 3, respectively. Let N represent the set of nodes in thehierarchical topology 115. For each node n in the set of nodes N (i.e.,∀nεN), let D(n) denote the set of n's descendants, which correspond tothe set of zero or more nodes (e.g., referred to as descendent nodes)located at lower hierarchical levels that can be reached from the node nby traversing one or more edges (e.g., via zero or more interveningnodes). Also, for each node n in the set of nodes N (i.e., ∀nεN), letA(n) denote the set of n's ancestors, which correspond to the set ofzero or more nodes (e.g., referred to as ancestor nodes) located athigher hierarchical levels that can be reached from the node n bytraversing one or more edges (e.g., via zero or more intervening nodes).Additionally, for each node n in the set of nodes N (i.e., ∀nεN), letd(n) denote the set of n's direct descendants, which are the set of zeroor more descendent nodes that can be reached from the node n bytraversing only one edge. Furthermore, for each node n in the set ofnodes N (i.e., ∀nεN), let a(n) denote the set of n's direct ancestors,which are the set of zero or more ancestor nodes that can be reachedfrom the node n by traversing only one edge. Using the foregoingformulation, the reported status 130 for the hierarchical topology 115is assumed to have the following two properties, referred to as propertyP1 and property P2:

Property P1: The reported status 130 for a particular node n indicatesthat the node is at least one of abnormal, normal or indeterminate.Property P1 can be stated mathematically as follows. Let f(n) denote thereported status for node n. Then, in some examples, for each node n inthe set of nodes N (i.e., ∀nεN), the reported status f(n) is given byEquation 1:

$\begin{matrix}{{\forall{n \in {N\text{:}{f(n)}}}} = \left\{ \begin{matrix}1 & {{if}\mspace{14mu} n\mspace{14mu} {is}\mspace{14mu} {abnomal}} \\0 & {{if}\mspace{14mu} n\mspace{14mu} {is}\mspace{14mu} {normal}} \\{- 1} & {{if}\mspace{14mu} n\mspace{20mu} {is}\mspace{14mu} {indeterminate}}\end{matrix} \right.} & {{Equation}\mspace{14mu} 1}\end{matrix}$

Property P2: Each abnormal node n identified in the reported status 130(e.g., each node n with a reported status 130 of f(n)=1), has at leastone abnormal or indeterminate descendent node (e.g., at least onedescendent node x with a reported status 130 of f(x)=1 or f(x)=−1).Property P2 can be represented mathematically using Equation 2:

∀nεN:f(n)=1

∃xεD(n):f(x)=1Vf(x)=−1  Equation 2

Based on the foregoing problem formulation, the example anomalylocalizer 135 of FIG. 4 determines a smallest subset A of the set ofnodes N in the hierarchical topology 115 (i.e., stated mathematically aschoosing A such that arg min |A|) that covers the abnormal

-   -   A⊂N        nodes identified in the reported status 130 (e.g., that covers        the nodes n having f(n)=1) subject to one or more constraints.        In some examples, the anomaly localizer 135 determines the        subset A that satisfies the following three constraints,        referred to as constraint C1, constraint C2 and constraint C3:

Constraint C1: Each node a in A must be abnormal. Constraint C1 can berepresented mathematically using Equation 3:

∀aεA:f(a)=1  Equation 3

Constraint C2: The subset A covers all of the abnormal nodes in N. Inother words, each abnormal node in N is either in A, or is a descendentof a node in A, or is an ancestor of a node in A. Constraint C2 can berepresented mathematically using Equation 4:

∀nεN:f(n)=1

∃aεA:n=a

nεD(a)

nεA(a)  Equation 4

Constraint C3: For any node a in A, the number of a's direct abnormaland indeterminate descendants is larger than the number of a's directnormal descendants. Constraint C3 can be represented mathematicallyusing Equation 5:

∀aεA:|{xεd(a)|f(x)=1

f(x)=−1}|>|{xεd(a)|f(x)=0}|  Equation 5

The subset A determined by the anomaly localizer 135 is referred to asthe subset of root cause abnormal nodes 140 that cover the abnormalnodes identified in the reported status 130. As such, each abnormal nodea included in the determined subset A is referred to as a root causeabnormal node.

In view of the foregoing formulation of the localization processingperformed by the anomaly localizer 135, the example anomaly localizer135 of FIG. 4 includes an example abnormal node identifier 405 to obtainthe reported status 130 (e.g., from the anomaly detector 105) andidentify the abnormal nodes of the hierarchical topology 115 indicatedby the reported status 130. For example, the abnormal node identifier405 can identify each abnormal node of the hierarchical topology 115 tobe those nodes n having a reported status 130 of f(n)=1.

The example anomaly localizer 135 of FIG. 4 further includes an exampleroot cause candidate selector 410 and an example root cause determiner415 to implement a greedy algorithm for determining the subset of rootcause abnormal nodes 140 (denoted by A above) that cover the abnormalnodes identified by the abnormal node identifier 405. In the illustratedexample, the root cause candidate selector 410 selects a set ofcandidate root cause abnormal nodes (denoted as U) such that each node uin the candidate set U satisfies Constraints 1 and 3 described above. Inother words, the root cause candidate selector 410 selects each abnormalnode identified by the abnormal node identifier 405 and that has anumber of direct abnormal descendents and direct indeterminatedescendents that is greater than the number of its direct normaldescendents for inclusion in the candidate set U.

In the illustrated example, the root cause candidate selector 410provides the candidate set U to the root cause determiner 415. The rootcause determiner 415 selects the subset of root cause nodes A from thecandidate set U such that the root cause subset A satisfies Constraint 2described above. Furthermore, the root cause determiner 415 performs agreedy selection that attempts to yield the root cause subset A thatcontains the smallest number of candidate nodes c from the candidate setU. An example operation 500 of the root cause determiner 415 todetermine the root cause subset A for an example candidate set U isillustrated in FIG. 5.

Turning to FIG. 5, in the illustrated example operation 500 of the rootcause determiner 415, the set of abnormal nodes includes nine (9)abnormal nodes u₁ through u₉ for an example hierarchical topology 115.The example operation 500 also illustrates four example subsets S₁through S₄ of the abnormal nodes u₁ through u₉ representing differentgroups of the candidate nodes covered in accordance with Constraint 2 bydifferent candidate nodes satisfying Constraints 1 and 3 describedabove. For example, using set-covering terminology, all the abnormalnodes in the hierarchical topology 115 form a universe. For a particularcandidate root cause abnormal node x from the set of abnormal nodes u₁through u₉ satisfying Constraints C1 and C3, a respective subset S_(x)of the universe is formed by the root cause determiner 415. The subsetS_(x) for a particular candidate root cause abnormal node x includes xitself, as well as all of x's descendents and ancestors in thehierarchical topology 115 (e.g., S_(x)=x∪D(x)∪A(x)). The result is afamily of n subsets (S₁, S₂, . . . , S_(n)) that correspond to the nabnormal nodes that satisfy the Constraints C1 and C3. The root causedeterminer 415 implements a greedy algorithm to determine the smallestsubfamily (or subset) of the n subsets (S₁, S₂, . . . , S_(n)) thatcover all of the abnormal nodes in the hierarchical topology 115. Forexample, the root cause determiner 415 iteratively chooses candidateroot cause abnormal nodes x that cover most uncovered abnormal nodesuntil all abnormal nodes of the hierarchical topology 115 are covered.In the illustrated example operation 500, the root cause determiner 415determines that the four subsets S₁ through S₄ cover all of the abnormalnodes of the hierarchical topology 115. Accordingly, the root causedeterminer 415 selects the candidate root cause abnormal nodes yieldingthe four subsets S₁ through S₄ of covered abnormal nodes for inclusionin the root cause abnormal set A.

Example machine readable instructions that may be used to implement theanomaly localizer 135 are illustrated in FIGS. 7-10, which are describedin greater detail below. In particular, FIGS. 9 and 10 illustrateexample machine readable instructions to perform a greedy procedure forroot cause abnormal node selection that may be used to implement theroot cause determiner 415.

A block diagram of an example implementation of the anomaly prioritizer145 of FIG. 1 is illustrated in FIG. 6. The anomaly prioritizer 145 ofthe illustrated example obtains a subset of root cause abnormal nodes140 (e.g., from the anomaly localizer 135) covering the abnormal nodesdetected in a hierarchical topology 115 and determines a ranking 150 ofthe root cause abnormal nodes. In the illustrated example of FIG. 6, theanomaly prioritizer 145 includes an example anomaly size determiner 605to determine a size of an anomaly (e.g., such as an anomaly event)associated with a root cause abnormal node included in the subset ofroot cause abnormal nodes 140. For example, if RTT is used for anomalydetection, then the anomaly size determined by the anomaly sizedeterminer 605 for a root cause abnormal node can correspond to adeviation or variance from an expected RTT at that node in thehierarchical topology, or an RTT magnitude determine at the node in thehierarchical topology, etc. Additionally or alternatively, the anomalysize determiner 605 can determine the deviation, variance, magnitude,degree, etc., of one or more other parameters used to determine that aroot cause abnormal node is associated with an anomalous event,anomalous behavior, etc.

The example anomaly prioritizer 145 of FIG. 6 also includes an exampleanomaly scope determiner 610 to determine a scope of an anomaly (e.g.,such as an anomaly event) associated with a root cause abnormal nodeincluded in the subset of root cause abnormal nodes 140. For example,the anomaly scope determined by the anomaly scope determiner 610 maycorrespond to a number of users, network elements, Internet protocoladdresses, prefixes, etc., affected by the anomaly associated with theroot cause abnormal node. In some example, such as when the hierarchicaltopology corresponds to a communication network, the scope (or breadth)of an anomaly associated with a node at a higher level in thehierarchical topology is generally larger than the scope (or breadth) ofan anomaly associated with a node at a lower level in the hierarchicaltopology. This is because nodes at higher hierarchical levels generallycorrespond to an aggregation of nodes at one or more lower hierarchicallevels.

The example anomaly prioritizer 145 of FIG. 6 further includes anexample anomaly rank evaluator 615 to process anomaly sizes and scopesdetermined by the anomaly size determiner 605 and the anomaly scopedeterminer 610, respectively, for the root cause abnormal nodes todetermine the ranking 150 of the set of root cause abnormal nodes 140.In some examples, the anomaly rank evaluator 615 implements a rankingfunction that multiplies the anomaly size for a particular root causeabnormal node and the anomaly scope for the particular root causeabnormal node to determine a ranking value for the particular root causeabnormal node. Other combining operations, such as addition of theanomaly sizes and scopes, normalization of the anomaly sizes and scopesby normalizing factors, etc., may additionally or alternatively be usedby the anomaly rank evaluator 615 to combine the anomaly sizes andscopes determined by the anomaly size determiner 605 and the anomalyscope determiner 610. In some examples, the anomaly rank evaluator 615ranks the root cause abnormal nodes included in the set of root causeabnormal nodes 140 based on the ranking values determined by combiningthe anomaly sizes and scopes. The anomaly rank evaluator 615 thenoutputs this ranking 150, which may be represented by an ordering of theroot cause abnormal nodes included in the set of root cause abnormalnodes 140, a numerical rank associated with each of the root causeabnormal nodes included in the set of root cause abnormal nodes 140,etc. Additionally or alternatively, the anomaly rank evaluator 615 canreport the raw ranking value determined for each root cause abnormalnode by combining the respective anomaly scopes and sizes.

While example manners of implementing the system 100 have beenillustrated in FIGS. 1, 4 and 6, one or more of the elements, processesand/or devices illustrated in FIGS. 1, 4 and/or 6 may be combined,divided, re-arranged, omitted, eliminated and/or implemented in anyother way. Further, the example anomaly detector 105, the example userworkstation 110, the example anomaly localizer 135, the example anomalyprioritizer 145, the example abnormal node identifier 405, the exampleroot cause candidate selector 410, the example root cause determiner415, the example anomaly size determiner 605, the example anomaly scopedeterminer 610, the example anomaly rank evaluator 615 and/or, moregenerally, the example system 100 may be implemented by hardware,software, firmware and/or any combination of hardware, software and/orfirmware. Thus, for example, any of the example anomaly detector 105,the example user workstation 110, the example anomaly localizer 135, theexample anomaly prioritizer 145, the example abnormal node identifier405, the example root cause candidate selector 410, the example rootcause determiner 415, the example anomaly size determiner 605, theexample anomaly scope determiner 610, the example anomaly rank evaluator615 and/or, more generally, the example system 100 could be implementedby one or more circuit(s), programmable processor(s), applicationspecific integrated circuit(s) (ASIC(s)), programmable logic device(s)(PLD(s)) and/or field programmable logic device(s) (FPLD(s)), etc. Whenany of the appended apparatus or system claims are read to cover apurely software and/or firmware implementation, at least one of theexample system 100, the example anomaly detector 105, the example userworkstation 110, the example anomaly localizer 135, the example anomalyprioritizer 145, the example abnormal node identifier 405, the exampleroot cause candidate selector 410, the example root cause determiner415, the example anomaly size determiner 605, the example anomaly scopedeterminer 610 and/or the example anomaly rank evaluator 615 are herebyexpressly defined to include a tangible computer readable medium such asa memory, digital versatile disk (DVD), compact disk (CD), etc., storingsuch software and/or firmware. Further still, the example system 100,the example anomaly localizer 135 and/or the example anomaly prioritizer145 of FIGS. 1, 4 and 6, respectively, may include one or more elements,processes and/or devices in addition to, or instead of, thoseillustrated in FIGS. 1, 4 and/or 6, and/or may include more than one ofany or all of the illustrated elements, processes and devices.

Flowcharts representative of example machine readable instructions thatmay be executed to implement the example system 100, the example anomalydetector 105, the example user workstation 110, the example anomalylocalizer 135, the example anomaly prioritizer 145, the example abnormalnode identifier 405, the example root cause candidate selector 410, theexample root cause determiner 415, the example anomaly size determiner605, the example anomaly scope determiner 610 and/or the example anomalyrank evaluator 615 are shown in FIGS. 7-11. In these examples, themachine readable instructions represented by each flowchart may compriseone or more programs for execution by a processor, such as the processor1212 shown in the example processing system 1200 discussed below inconnection with FIG. 12. Alternatively, the entire program or programsand/or portions thereof implementing one or more of the processesrepresented by the flowcharts of FIGS. 7-11 could be executed by adevice other than the processor 1212 (e.g., such as a controller and/orany other suitable device) and/or embodied in firmware or dedicatedhardware (e.g., implemented by an ASIC, a PLD, an FPLD, discrete logic,etc.). Also, one or more of the machine readable instructionsrepresented by the flowchart of FIGS. 7-11 may be implemented manually.Further, although the example machine readable instructions aredescribed with reference to the flowcharts illustrated in FIGS. 7-11,many other techniques for implementing the example methods and apparatusdescribed herein may alternatively be used. For example, with referenceto the flowcharts illustrated in FIGS. 7-11, the order of execution ofthe blocks may be changed, and/or some of the blocks described may bechanged, eliminated, combined and/or subdivided into multiple blocks.

As mentioned above, the example processes of FIGS. 7-11 may beimplemented using coded instructions (e.g., computer readableinstructions) stored on a tangible computer readable medium such as ahard disk drive, a flash memory, a read-only memory (ROM), a CD, a DVD,a cache, a random-access memory (RAM) and/or any other storage media inwhich information is stored for any duration (e.g., for extended timeperiods, permanently, brief instances, for temporarily buffering, and/orfor caching of the information). As used herein, the term tangiblecomputer readable medium is expressly defined to include any type ofcomputer readable storage and to exclude propagating signals.Additionally or alternatively, the example processes of FIGS. 7-11 maybe implemented using coded instructions (e.g., computer readableinstructions) stored on a non-transitory computer readable medium, suchas a flash memory, a ROM, a CD, a DVD, a cache, a random-access memory(RAM) and/or any other storage media in which information is stored forany duration (e.g., for extended time periods, permanently, briefinstances, for temporarily buffering, and/or for caching of theinformation). As used herein, the term non-transitory computer readablemedium is expressly defined to include any type of computer readablemedium and to exclude propagating signals. Also, as used herein, theterms “computer readable” and “machine readable” are consideredequivalent unless indicated otherwise.

Example machine readable instructions 700 that may be executed toimplement the system 100 of FIG. 1 are represented by the flowchartshown in FIG. 7. With reference to the preceding figures, the machinereadable instructions 700 of FIG. 7 begin execution at block 705 atwhich the user workstation 110 is used to obtain configurationinformation, such as the hierarchical topology 115 for which anomalydetection, localization and prioritization is to be performed.Additionally or alternatively, at block 705 the user workstation 110 canbe used to obtain the other configuration information 120 used in thesystem 100.

At block 710, the anomaly detector 105 processes the measurementinformation 125 to detect anomalies associated with one or more nodes ofthe hierarchical topology 115. As described above, the anomaly detector105 can employ any type of anomaly detection processing. At block 710,the anomaly detector 105 also provides reported status 130 for the nodesin the hierarchical topology 115. For example, the reported status 130can use a value of “1” to indicate an abnormal node, a value of “0” toindicate a normal node, and a value of “−1” to indicate an indeterminatenode. Other techniques for representing the status of the nodes in thehierarchical topology 115 may additionally or alternatively be used.

At block 715, the anomaly localizer 135 uses the reported status 130 forthe hierarchical topology 115 to perform anomaly localization. Forexample, the anomaly localizer 135 can identify the abnormal nodesindicated by the reported status 130 and determine a subset of rootcause abnormal nodes 140 that cover (e.g., account for, explain, etc.)all, or a portion of, the abnormal nodes indicated by the reportedstatus 130. Example machine readable instructions that may be used toimplement the processing at block 715 are illustrated in FIG. 8, whichis described in greater detail below.

At block 720, the anomaly prioritizer 145 prioritizes the root causeabnormal nodes included in the subset of root cause abnormal nodes 140determined at block 715. For example, the anomaly prioritizer 145 canrank the root cause abnormal nodes based on one or more parameters, suchas an anomaly size, and anomaly scope, etc., or a combination thereof.Example machine readable instructions that may be used to implement theprocessing at block 720 are illustrated in FIG. 11, which is describedin greater detail below.

Example machine readable instructions 715 that may be executed toimplement the anomaly localizer 135 of FIGS. 1 and 4 are represented bythe flowchart shown in FIG. 8. With reference to the preceding figures,the machine readable instructions 715 of FIG. 8 begin execution at block805 at which the anomaly localizer 135 obtains the hierarchical topology115 for which anomaly localization is to be performed. For example, thehierarchical topology 115 may be specified by the user workstation 110,pre-configured, etc., or any combination thereof. The hierarchicaltopology 115 may be represented using any data structure, format, file,etc., or combination thereof capable of describing nodes, edges,descendent relationships, ancestor relationships, etc.

At block 810, the abnormal node identifier 405 of the anomaly localizer135 obtains the reported status 130 for the nodes included in thehierarchical topology 115. For example, the reported status 130 can usea value of “1” to indicate an abnormal node, a value of “0” to indicatea normal node, a value of “−1” to indicate an indeterminate node, etc.Other techniques for representing the status of the nodes in thehierarchical topology 115 may additionally or alternatively be used.

At block 815, the abnormal node identifier 405 uses the reported status130 obtained at block 810 to identify the abnormal nodes detected in thehierarchical topology 115. For example, the abnormal node identifier 405can identify the abnormal nodes to be those nodes having a reportedstatus set to “1” or another value representative of an abnormal status.At block 820, the root cause candidate selector 410 of the anomalylocalizer 135 selects a set of candidate root cause abnormal nodes suchthat each node in the candidate set satisfies Constraints 1 and 3described above. In other words, at block 820, the root cause candidateselector 410 selects a set of candidate root cause abnormal nodes suchthat each candidate root cause node in the set is an abnormal node andhas a number of direct abnormal and indeterminate descendants that islarger than the number of its direct normal descendants.

At block 825, the root cause determiner 415 of the anomaly localizer 135determines a subset of root cause abnormal nodes from the set ofcandidate root cause abnormal nodes selected at block 820. For example,at block 825 the root cause determiner 415 selects the subset of rootcause nodes 140 from the candidate set such that the root cause subset140 satisfies Constraint 2 described above and, thus, covers all, or aportion of, the abnormal nodes of the hierarchical topology 115. In someexamples, the root cause determiner 415 implements a greedy algorithmthat attempts to select the smallest subset of root cause nodes 140 thatcovers all of the abnormal nodes identified in the hierarchical topology115. Example machine readable instructions that may be used to implementthe processing at block 825 are illustrated in FIG. 9, which isdescribed in greater detail below.

Example machine readable instructions 825 to perform an example greedyalgorithm to implement the root cause determiner 415 of the anomalylocalizer 135 of FIG. 4 are represented by the flowchart shown in FIG.9. Example pseudocode 1000 that may be used to implement the machinereadable instructions 825 is illustrated in FIG. 10. For simplicity andclarity, the example pseudocode 1000 of FIG. 10 is described togetherwith the description of the example machine readable instructions 825 ofFIG. 9. As such, and with reference to the preceding figures, themachine readable instructions 825 of FIG. 9 begin execution at block 905at which the root cause determiner 415 initializes the set of root causeabnormal nodes 140 to be empty (e.g., the null set). The processing atblock 905 corresponds to section 1005 of the pseudocode 1000 of FIG. 10.

At block 910 of FIG. 9, the root cause determiner 415 initializes a setrepresenting the uncovered abnormal nodes to include all identifiedabnormal nodes of the hierarchical topology 115. The processing at block910 also corresponds to section 1005 of the pseudocode 1000 of FIG. 10.The goal of the greedy algorithm implemented by the root causedeterminer 415 is to iteratively select candidate root cause abnormalnodes to determine a smallest subset that covers the abnormal nodesincluded in the uncovered set. Each time a candidate root cause abnormalnode is selected to be a root cause abnormal node for inclusion in thesubset 140, the abnormal nodes covered by this selected root causeabnormal node are removed from the uncovered set.

At block 915, the root cause determiner 415 determines, for eachcandidate root cause abnormal node u (e.g., each abnormal nodesatisfying Constraints 1 and 3 described above), a respective subset ofabnormal nodes SET_(u) (also denoted by S_(u) above) that are covered bythe respective candidate root cause abnormal node u. The processing atblock 910 corresponds to section 1010 of the pseudocode 1000 of FIG. 10.In particular, pseudocode section 1010 includes pseudocode section 1015in which the subset of abnormal nodes SET, for the candidate root causeabnormal node u is initialized to include the candidate node u itself.Pseudocode section 1020 included in pseudocode section 1010 adds thedescendent abnormal nodes covered by the candidate node u to its coverednode subset SET_(u). Pseudocode section 1025 included in pseudocodesection 1010 adds the ancestor abnormal nodes covered by the candidatenode u to its covered node subset SET_(u).

At block 920 of FIG. 9, the root cause determiner 415 selects thecandidate root cause abnormal node u remaining in the uncovered set ofabnormal nodes that has largest covered node subset SET_(u). In otherwords, at block 920 the root cause determiner 415 selects a nextcandidate root cause abnormal node u that covers the largest subset ofabnormal nodes that have not yet been covered by a previously selectedcandidate root cause abnormal node. The processing at block 920corresponds to pseudocode section 1030 included in the pseudocode 1000of FIG. 10.

At block 925 of FIG. 9, the root cause determiner 415 includes thecandidate root cause abnormal node u selected at block 920 as a rootcause abnormal node in the subset of root cause abnormal nodes 140. Theprocessing at block 925 corresponds to pseudocode section 1035 includedin the pseudocode 1000 of FIG. 10. At block 930 of FIG. 9, the rootcause determiner 415 removes the subset SET_(u) of abnormal nodescovered by the selected candidate root cause abnormal node u from theuncovered set (because these abnormal nodes are now covered by a rootcause abnormal node in the subset of root cause abnormal nodes 140). Theprocessing at block 930 corresponds to pseudocode section 1040 includedin the pseudocode 1000 of FIG. 10. Additionally, at block 935 the rootcause determiner 415 removes the subset SET_(u) of abnormal nodescovered by the selected candidate root cause abnormal node u from theother covered subsets corresponding to the other remaining candidateroot cause abnormal nodes. The processing at block 930 corresponds topseudocode section 1045 included in the pseudocode 1000 of FIG. 10.

At block 940, the root cause determiner 415 determines whether theuncovered set of abnormal nodes is empty (e.g., because all abnormalnodes are covered by candidate nodes that have been selected to be rootcause abnormal nodes included in the subset 140). If the uncovered setis not empty (block 940), processing returns to block 920 and blockssubsequent thereto to enable the root cause determiner 415 to select anext candidate root cause abnormal node u for inclusion in the subset ofroot cause abnormal nodes 140. However, if the uncovered set is empty(block 940), then at block 945 the root cause determiner 415 outputs thedetermined subset of root cause abnormal nodes 140.

Example machine readable instructions 720 that may be executed toimplement the anomaly prioritizer 145 of FIGS. 1 and 6 are representedby the flowchart shown in FIG. 11. With reference to the precedingfigures, the machine readable instructions 720 of FIG. 11 beginexecution at block 1105 at which the anomaly prioritizer 145 obtains thesubset of root cause abnormal nodes 140 that covers the abnormal nodesdetected in a hierarchical topology 115. At block 1110, the anomaly sizedeterminer 605 of the anomaly prioritizer 145 determines sizes of theanomalies associated with the respective root cause abnormal nodesincluded in the subset 140 obtained at block 1105. At block 1115, theanomaly scope determiner 610 of the anomaly prioritizer 145 determinesscopes of the anomalies associated with the respective root causeabnormal nodes included in the subset 140 obtained at block 1105. Atblock 1120, the anomaly rank evaluator 615 of the anomaly prioritizer145 combines the anomaly sizes determined at block 1110 and the anomalyscopes determined at block 1115 to determine a ranking 150 of the rootcause abnormal nodes included in the subset 140.

FIG. 12 is a block diagram of an example processing system 1200 capableof implementing the apparatus and methods disclosed herein. Theprocessing system 1200 can be, for example, a server, a personalcomputer, a tablet computer, a personal digital assistant (PDA), anInternet appliance, or any other type of computing device.

The system 1200 of the instant example includes a processor 1212 such asa general purpose programmable processor. The processor 1212 includes alocal memory 1214, and executes coded instructions 1216 present in thelocal memory 1214 and/or in another memory device. The processor 1212may execute, among other things, the machine readable instructionsrepresented in FIGS. 7-11. The processor 1212 may be any type ofprocessing unit, such as one or more Intel® microprocessors from thePentium® family, the Itanium® family and/or the XScale® family, one ormore microcontrollers from the ARM® and/or PIC® families ofmicrocontrollers, etc. Of course, other processors from other familiesare also appropriate.

The processor 1212 is in communication with a main memory including avolatile memory 1218 and a non-volatile memory 1220 via a bus 1222. Thevolatile memory 1218 may be implemented by Static Random Access Memory(SRAM), Synchronous Dynamic Random Access Memory (SDRAM), Dynamic RandomAccess Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM) and/orany other type of random access memory device. The non-volatile memory1220 may be implemented by flash memory and/or any other desired type ofmemory device. Access to the main memory 1218, 1220 is typicallycontrolled by a memory controller (not shown).

The processing system 1200 also includes an interface circuit 1224. Theinterface circuit 1224 may be implemented by any type of interfacestandard, such as an Ethernet interface, a universal serial bus (USB),and/or a third generation input/output (3GIO) interface.

One or more input devices 1226 are connected to the interface circuit1224. The input device(s) 1226 permit a user to enter data and commandsinto the processor 1212. The input device(s) can be implemented by, forexample, a keyboard, a mouse, a touchscreen, a track-pad, a trackball,an isopoint and/or a voice recognition system.

One or more output devices 1228 are also connected to the interfacecircuit 1224. The output devices 1228 can be implemented, for example,by display devices (e.g., a liquid crystal display, a cathode ray tubedisplay (CRT)), by a printer and/or by speakers. The interface circuit1224, thus, typically includes a graphics driver card.

The interface circuit 1224 also includes a communication device such asa modem or network interface card to facilitate exchange of data withexternal computers via a network (e.g., an Ethernet connection, adigital subscriber line (DSL), a telephone line, coaxial cable, acellular telephone system, etc.).

The processing system 1200 also includes one or more mass storagedevices 1230 for storing machine readable instructions and data.Examples of such mass storage devices 1230 include floppy disk drives,hard drive disks, compact disk drives and digital versatile disk (DVD)drives.

The coded instructions 1232 of FIGS. 7-11 may be stored in the massstorage device 1230, in the volatile memory 1218, in the non-volatilememory 1220, in the local memory 1214 and/or on a removable storagemedium, such as a CD or DVD 1232.

At least some of the above described example methods and/or apparatusare implemented by one or more software and/or firmware programs runningon a computer processor. However, dedicated hardware implementationsincluding, but not limited to, application specific integrated circuits,programmable logic arrays and other hardware devices can likewise beconstructed to implement some or all of the example methods and/orapparatus described herein, either in whole or in part. Furthermore,alternative software implementations including, but not limited to,distributed processing or component/object distributed processing,parallel processing, or virtual machine processing can also beconstructed to implement the example methods and/or apparatus describedherein.

To the extent the above specification describes example components andfunctions with reference to particular standards and protocols, it isunderstood that the scope of this patent is not limited to suchstandards and protocols. For instance, each of the standards forInternet and other packet switched network transmission (e.g.,Transmission Control Protocol (TCP)/Internet Protocol (IP), UserDatagram Protocol (UDP)/IP, HyperText Markup Language (HTML), HyperTextTransfer Protocol (HTTP)) represent examples of the current state of theart. Such standards are periodically superseded by faster or moreefficient equivalents having the same general functionality.Accordingly, replacement standards and protocols having the samefunctions are equivalents which are contemplated by this patent and areintended to be included within the scope of the accompanying claims.

Additionally, although this patent discloses example systems includingsoftware or firmware executed on hardware, it should be noted that suchsystems are merely illustrative and should not be considered aslimiting. For example, it is contemplated that any or all of thesehardware and software components could be embodied exclusively inhardware, exclusively in software, exclusively in firmware or in somecombination of hardware, firmware and/or software. Accordingly, whilethe above specification described example systems, methods and articlesof manufacture, the examples are not the only way to implement suchsystems, methods and articles of manufacture. Therefore, althoughcertain example methods, apparatus and articles of manufacture have beendescribed herein, the scope of coverage of this patent is not limitedthereto. On the contrary, this patent covers all methods, apparatus andarticles of manufacture fairly falling within the scope of the claimseither literally or under the doctrine of equivalents.

1. A method comprising: obtaining reported status for a plurality ofnodes of a hierarchical topology, the reported status for a particularnode being at least one of normal, abnormal or indeterminate; anddetermining with a processor a subset of root cause abnormal nodes, aroot cause abnormal node included in the subset of root cause abnormalnodes having a total number of abnormal direct descendent nodes andindeterminate direct descendent nodes that is greater than a number ofnormal direct descendent nodes of the root cause abnormal node.
 2. Amethod as defined in claim 1 wherein each node of the hierarchicaltopology represents a respective hierarchical portion of a network, andthe reported status is based on measured performance of the network. 3.A method as defined in claim 1 wherein the subset of root cause abnormalnodes covers all abnormal nodes in the hierarchical topology.
 4. Amethod as defined in claim 1 wherein determining the subset of rootcause abnormal nodes comprises: selecting a set of candidate root causeabnormal nodes from a set of abnormal nodes in the hierarchicaltopology; and determining the subset of root cause abnormal nodes fromthe set of candidate root cause abnormal nodes.
 5. A method as definedin claim 4 wherein a candidate root cause abnormal node included in theset of candidate root cause abnormal nodes has a respective number ofabnormal direct descendent nodes and indeterminate direct descendentnodes that is greater than a respective number of normal directdescendent nodes of the candidate root cause abnormal node.
 6. A methodas defined in claim 4 wherein determining the subset of root causeabnormal nodes from the set of candidate root cause abnormal nodescomprises: determining for each respective candidate root cause abnormalnode a respective subset of the set of abnormal nodes in thehierarchical topology covered by the respective candidate root causeabnormal node; and selecting a first candidate root cause abnormal nodefor inclusion in the subset of root cause abnormal nodes if the firstcandidate root cause abnormal node covers a largest subset of the set ofabnormal nodes in the hierarchical topology.
 7. A method as defined inclaim 6 wherein the respective subset of the set of abnormal nodescovered by the respective candidate root cause abnormal node includesany abnormal node that is a descendent or an ancestor of the respectivecandidate root cause abnormal node in the hierarchical topology.
 8. Atangible article of manufacture storing machine readable instructionswhich, when executed, cause a machine to at least: obtain reportedstatus for a plurality of nodes of a hierarchical topology, the reportedstatus for a particular node being at least one of normal, abnormal orindeterminate; and determining a subset of root cause abnormal nodes, aroot cause abnormal node included in the subset of root cause abnormalnodes having a total number of abnormal direct descendent nodes andindeterminate direct descendent nodes that is greater than a number ofnormal direct descendent nodes of the root cause abnormal node.
 9. Atangible article of manufacture as defined in claim 8 wherein each nodeof the hierarchical topology represents a respective hierarchicalportion of a network, and the reported status is based on measuredperformance of the network.
 10. A tangible article of manufacture asdefined in claim 8 wherein the subset of root cause abnormal nodescovers all abnormal nodes in the hierarchical topology.
 11. A tangiblearticle of manufacture as defined in claim 8 wherein to determine thesubset of root cause abnormal nodes, the machine readable instructions,when executed, further cause the machine to: select a set of candidateroot cause abnormal nodes from a set of abnormal nodes in thehierarchical topology; and determine the subset of root cause abnormalnodes from the set of candidate root cause abnormal nodes.
 12. Atangible article of manufacture as defined in claim 11 wherein acandidate root cause abnormal node included in the set of candidate rootcause abnormal nodes has a respective number of abnormal directdescendent nodes and indeterminate direct descendent nodes that isgreater than a respective number of normal direct descendent nodes ofthe candidate root cause abnormal node.
 13. A tangible article ofmanufacture as defined in claim 11 wherein to determine the subset ofroot cause abnormal nodes from the set of candidate root cause abnormalnodes, the machine readable instructions, when executed, further causethe machine to: determine for each respective candidate root causeabnormal node a respective subset of the set of abnormal nodes in thehierarchical topology covered by the respective candidate root causeabnormal node; and select a first candidate root cause abnormal node forinclusion in the subset of root cause abnormal nodes if the firstcandidate root cause abnormal node covers a largest subset of the set ofabnormal nodes in the hierarchical topology.
 14. A tangible article ofmanufacture as defined in claim 13 wherein the respective subset of theset of abnormal nodes covered by the respective candidate root causeabnormal node includes any abnormal node that is a descendent or anancestor of the respective candidate root cause abnormal node in thehierarchical topology.
 15. An apparatus comprising: a candidate selectorto select a set of candidate root cause abnormal nodes from a set ofabnormal nodes identified in a hierarchical topology, a particularcandidate root cause abnormal node having a total number of abnormaldirect descendent nodes and indeterminate direct descendent nodes thatis greater than a number of normal direct descendent nodes of theparticular candidate root cause abnormal node; and a root causedeterminer to determine a set of root cause abnormal nodes from the setof candidate root cause abnormal nodes selected by the candidateselector, the set of root cause abnormal nodes to cover the set ofabnormal nodes identified in the hierarchical topology.
 16. An apparatusas defined in claim 15 wherein each node of the hierarchical topologyrepresents a respective hierarchical portion of a network, and each nodeof the hierarchical topology is classified to be at least one of normal,abnormal or indeterminate based on measured performance of the network.17. An apparatus as defined in claim 15 wherein the root causedeterminer is to determine for each respective candidate root causeabnormal node a respective subset of the set of abnormal nodesidentified in a hierarchical topology that are covered by the respectivecandidate root cause abnormal node, the respective subset of the set ofabnormal nodes covered by the respective candidate root cause abnormalnode including any abnormal node that is a descendent or an ancestor ofthe respective candidate root cause abnormal node in the hierarchicaltopology.
 18. An apparatus as defined in claim 17 wherein the root causedeterminer is to select a first candidate root cause abnormal node forinclusion in the subset of root cause abnormal nodes if the firstcandidate root cause abnormal node covers a largest subset of the set ofabnormal nodes identified in a hierarchical topology.
 19. An apparatusas defined in claim 17 wherein the root cause determiner is toiteratively select candidate root cause abnormal nodes for inclusion inthe subset of root cause abnormal nodes based on sizes of the subsets ofthe set of abnormal nodes covered by the respective candidate root causeabnormal nodes.
 20. An apparatus as defined in claim 15 furthercomprising a prioritizer to: determine a size of a respectiveabnormality associated with each root cause abnormal node included inthe subset of root cause abnormal nodes; determine a scope for therespective abnormality associated with each root cause abnormal nodeincluded in the subset of root cause abnormal nodes; and rank the rootcause abnormal nodes included in the subset of root cause abnormal nodesbased on size and scope.